Data Dependent Distance Metric for Efficient Gaussian Processes Classification

The contributions of this work are threefold. First, various metric learning techniques are analyzed and systematically studied under a unified framework to highlight the criticality of data-dependent distance metric in machine learning. The metric learning algorithms are categorized as naive, semi-naive, complete and high-level metric learning, under a common distance measurement framework. Secondly, the connection of feature selection, feature weighting, feature partitioning, kernel tuning, etc. with metric learning is discussed and it is shown that they are all in fact forms of metric learning. Thirdly, it has been shown that the realm of metric learning is not limited to k-nearest neighbor (k-NN) classification, and that a metric optimized in the k-nearest neighbor setting is likely to be effective and applicable in other kernel-based frameworks, for example Support Vector Machine (SVM) and Gaussian Processes (GP) classifiers. We support our hypotheses by tuning the length-scale parameters of GP with metric learning method proposed in k-NN framework. Our empirical results on a huge range of machine learning databases suggest that a metric optimized in the framework of one learning algorithm is likely to be effective in those of others.